Scheduling and Traffic Management with Offload Processors

ABSTRACT

A scheduling system for a packet processing system is disclosed. The system can include a classification circuit connected to a memory bus and configurable to classify network packets, placing the classified network packets into first multiple input/output queues, a scheduling circuit for reordering the network packets received from the classification circuit through the first multiple input/output queues and placing the reordered network packets into second multiple input/output queues, an arbitration circuit for directing network packets received from the scheduling circuit through the second multiple input/output queues to multiple output ports, and multiple offload processors, each coupled to at least one of the multiple output ports, the offload processors configured to modify the network packets.

PRIORITY CLAIMS

This application claims the benefit of U.S. Provisional PatentApplications 61/753,892 filed on Jan. 17, 2013, 61/753,895 filed on Jan.17, 2013, 61/753,899 filed on Jan. 17, 2013, 61/753,901 filed on Jan.17, 2013, 61/753,903 filed on Jan. 17, 2013, 61/753,904 filed on Jan.17, 2013, 61/753,906 filed on Jan. 17, 2013, 61/753,907 filed on Jan.17, 2013, and 61/753,910 filed on Jan. 17, 2013, the contents all ofwhich are incorporated by reference herein.

TECHNICAL FIELD

Described embodiments relate to scheduling and traffic managementservices for computer systems that can be provided by a memory busconnected module with offload processors.

BACKGROUND

Efficient managing of network packet flow and processing is critical forhigh performance networked computing systems. Network packet flow can behighly variable, depending on hardware configurations, process flows anddata flows, with data processing needs varying over several orders ofmagnitude on time scales that can range from seconds to hours.Substantial improvements in network service are made possible by systemsthat can flexibly process a data flow, recognize or characterizepatterns in the data flow, and improve routing and processing decisionsfor the data flow. This is of particular importance for networkedcomputer environment using packet switching communication. For example,delays in data flow are often created due to network security requiredpacket inspection. Such packet inspection may be directed at either aheader of the packet or a payload of the packet, and can includeprocessor content matching, behavioral anomaly detection, “black” or“white” listing comparisons, or the like. Other high packet processingapplications can include encryption/decryption, quality of servicecontrolled packet reassembly, streaming sensor data, or video or audioprocessing. Without an efficient mechanism for scheduling packetprocessing arriving as part of a complex data flow system, users mayencounter unacceptable delays in network system response.

Commonly available traffic management circuits supporting a packetswitch fabric capable of handling complex data flow streams ofteninclude depth-limited output queues, the access to which is arbitratedby a scheduling circuit. The input queues are managed using a schedulingdiscipline to provide traffic management for incoming data flows.Schedulers may allocate or identify a data flow priorities and provideoutput port to each of these data flows. If multiple data flows competefor the same output port, time multiplexed access to each of the outputports can be provided, or alternatively multiple data flows contendingfor an output port may be arbitrated by an arbitration circuit beforebeing sent out over an output port. However, a traffic managementcircuit typically has limited or no access to information relating tohandling and management of data by downstream memory or processingelements. For example, based on an allocation of priority, data flowperformance can be improved if incoming packets can be dynamicallyreordered in a buffer to help maintain persistence of session flows inthese queues. The scheduling discipline chosen for such packetprocessing prioritization or traffic management (TM), can affect thetraffic shape of flows and micro-flows through delay (buffering),bursting of traffic (buffering and bursting), smoothing of traffic(buffering and rate-limiting flows), dropping traffic (choosing data todiscard so as to avoid exhausting the buffer), or delay jitter(temporally shifting cells of a flow by different amounts).

SUMMARY

This disclosure describes embodiments of systems, hardware and methodssuitable to act as a scheduling system for a packet processing system.The system includes a classification circuit connected to a memory busand configurable to classify network packets. The classification circuitcan place the classified network packets into first multipleinput/output queues. A scheduling circuit can reorder the networkpackets received from the classification circuit through the firstmultiple input/output queues and place the reordered network packetsinto second multiple input/output queues. An arbitration circuit directsnetwork packets received from the scheduling circuit through the secondmultiple input/output queues to multiple output ports. Multiple offloadprocessors, each connected to one of the multiple output ports, arerespectively configured to modify network packets.

In certain embodiments the memory bus supports direct memory access, andmultiple offload processors can direct modified packets back to thememory bus. In addition, the classification circuit can classify networkpackets based on session metadata. In still other embodiments, thescheduling circuit can direct network packets based on availability ofrespective multiple offload processors; reorder network packetsaccording to session priority; initiates a context switch for themultiple offload processors; transfer network packets into a definedtraffic management queue; check with each of the multiple offloadprocessors to determine if respective network packet processing iscomplete; or operate in preemption mode to control session execution.

Another embodiment is a method for scheduling packet processing,including the step of classifying network packets based on sessionmetadata and placing the classified network packets into first multipleinput/output queues, with packets transported to a classificationcircuit using a memory bus having a defined memory transport protocol.Reordered network packets received from the first multiple input/outputqueues using a scheduling circuit can be placed into a second multipleinput/output queues, where an arbitration circuit directs networkpackets received from the scheduling circuit through the second multipleinput/output queues into multiple output ports. These network packetscan be modified using multiple offload processors, each offloadprocessor respectively connected to one of the multiple output ports,with the respective offload processers able to direct modified packetsback to the memory bus.

Another embodiment described includes a memory bus connected module forscheduling services for network packet processing. The module includes amemory bus connection and a scheduling circuit for reordering thenetwork packets received from the memory bus connection and placing thereordered network packets into multiple input/output queues. Multipleoffload processors are connected to the memory bus connection, with eachoffload processor capable of modifying network packets placed intomultiple input/output queues. The memory bus connection can becompatible with a memory bus socket, and in certain embodiments beformed to fit into a dual in-line memory module (DIMM) socket.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1-0 shows a traffic management and scheduler system according to anembodiment.

FIG. 1-1 shows a scheduling process according to an embodiment.

FIG. 1-2 shows a module supporting multiple scheduling circuits and anarbitration circuit according to an embodiment.

FIGS. 2-0 to 2-3 show processor modules according to variousembodiments.

FIG. 2-4 shows a conventional dual-in-line memory module.

FIG. 2-5 shows a system according to another embodiment.

FIG. 3 shows one particular implementation of a memory bus connectedoffload processor that can be included in embodiments.

FIG. 4 shows an exemplary flow chart for a scheduling process accordingto an embodiment.

DETAILED DESCRIPTION

Various embodiments of the present invention will now be described indetail with reference to a number of drawings. The embodiments showprocessing systems and methods for scheduling packet flow in a packetprocessing systems. Such scheduling can be performed by, or with the useof, offload modules connect to a memory bus of a system. Such offloadprocessors can be in addition to any host processors connected to thesystem memory bus, and, in some embodiments, process packets transferredover the system memory bus independent of any host processors. In veryparticular embodiments, processing modules can populate physical slotsfor connecting in-line memory modules (e.g., DIMMs) to a system memorybus.

FIG. 1-0 is a diagram of a system 100 for providing scheduling andtraffic management services. A system 100 can include a switch 106, hostprocessor section 108/110, memory controller 112, and offload processingsection 116/116/118. In the particular embodiment shown, host processorsection can include a switch 106, a switching fabric 108, hostprocessor(s) 110, and a bus interconnect 109 connected to memorycontroller 112. Further, offload processing section can be incommunication with memory controller 112, and can include a switch 114,scheduler 116 and offload processor(s) 118.

In operation, a switch 106 can receive and/or transmit data packets 104from data source 102. A data source 102 can be any suitable source ofpacket data, including the Internet, a network cloud, inter- orintra-data center networks, cluster computers, rack systems, multiple orindividual servers or personal computers, or the like. Data can bepacket or switch based, although in particular embodiments non-packetdata is generally converted or encapsulated into packets for ease ofhandling. The data packets typically have certain characteristics,including transport protocol number, source and destination portnumbers, or source and destination (Internet Protocol) IP addresses. Thedata packets can further have associated metadata that helps in packetclassification and management.

A switch 106 can be a virtual switch (an I/O device). A switch 106 caninclude, but is not limited to, devices compatible with peripheralcomponent interconnect (PCI) and/or PCI express (PCIe) devicesconnecting with host motherboard via PCI or PCIe bus 107. The switch 106can include a network interface controller (NIC), a host bus adapter, aconverged network adapter, or a switched or an asynchronous transfermode (ATM) network interface. In some embodiments, a switch 106 canemploy IO virtualization schemes such as a single root I/Ovirtualization (SR-IOV) interface to make a single network I/O deviceappear as multiple devices. SR-IOV permits separate access to resourcesamong various PCIe hardware functions by providing both physical controland virtual functions. In certain embodiments, the switch 106 cansupport OpenFlow or similar software defined networking to abstract outof the control plane. The control plane of the first virtual switchperforms functions such as route determination, target nodeidentification etc.

A switch 106 can be capable of examining network packets, and using itscontrol plane to create appropriate output ports for network packets.Based on route calculation for the network packets or data flowsassociated with the network packets, the forwarding plane of the switch106 can transfer the packets to an output interface. An output interfaceof the switch may be connected with an IO bus, and in certainembodiments the switch 106 may have the capability to directly (orindirectly, via an I/O fabric 108) transfer the network packets to amemory bus interconnect 109 for a memory read or write operation (directmemory access operation). Functionally, for certain applications thenetwork packets can be assigned for transport to specific memorylocations based on control plane functionality.

Switch 106, connected to an IO fabric 108 and memory bus interconnect109, can also be connected to host processor(s) 110. Host processor(s)110 can include one or more host processors which can providecomputational services including a provisioning agent 111. Theprovisioning agent 111 can be part of an operating system or user coderunning on the host processor(s) 110. The provisioning agent 111typically initializes and interacts with virtual function driversprovided by system 100. The virtual function driver can be responsiblefor providing the virtual address of the memory space where a directmemory addressing (DMA) is needed. Each device driver can be allocatedvirtual addresses that map to the physical addresses. A device model canbe used to create an emulation of a physical device for the hostprocessor 110 to recognize each of the multiple virtual functions (VF)that can be created. The device model can be replicated multiple timesto give the impression to VF drivers (a driver that interacts with avirtual IO device) that they are interacting with a physical device. Forexample, a certain device model may be used to emulate a network adapterthat the VF driver can act to connect. The device model and the VFdriver can be run in either privileged or non-privileged mode. There canbe no restriction with regard to which device hosts/runs the codecorresponding to the device model and the VF driver. The code, however,can have the capability to create multiple copies of device model and VFdriver so as to enable multiple copies of said I/O interface to becreated. In certain embodiments the operating system can also create adefined physical address space for applications supported by VF drivers.Further, the host operating system can allocate a virtual memory addressspace to the application or provisioning agent. The provisioning agent111 can broker with the host operating system to create a mappingbetween virtual addresses and a subset of the available physical addressspace. The provisioning agent 111 can be responsible for creating eachVF driver and allocating it a defined virtual address space.

By operation of such memory mapping, data (e.g., packet data) can betransmitted from switch 106 offload processor section 114/116/118. Asecond switch 114 can also be connected to the memory controller 112 bymemory bus 109. A second switch 114 can be a virtual switch, and canreceive and switch traffic originating from the memory bus 109 both toand from offload processor(s) 118. Traffic may include, but is notlimited to, data flows to virtual devices created and assigned by theprovisioning agent 111, with processing supported by offload processors118. The forwarding plane of the second switch 114 can transportspackets from a memory bus 109 to offload processors 118 or from theoffload processors 118 back onto the memory bus 109. For certainapplications, the described system architecture can allow relativelydirect communication of network packets to the offload processors 118with minimal or no interruptions to a host processor(s) 110. The secondswitch 114 can be capable of receiving packets and classifying themprior to distribution to different hardware schedulers based on adefined arbitration and scheduling scheme. The hardware scheduler 116receives packets that can be assigned to flow sessions that arescheduled for processing in one or more separate sessions run by offloadprocessor(s) 118.

In particular embodiments, scheduler 116 can be employed to implementtraffic management of incoming packets. Packets from a certain source,relating to a certain traffic class, pertaining to a specificapplication or flowing to a certain socket are referred to as part of asession flow and are classified using session metadata. Session metadataoften serve as the criterion by which packets are prioritized and assuch, incoming packets can be reordered based on their session metadata.This reordering of packets can occur in one or more buffers and canmodify the traffic shape of these flows. Packets of a session that arereordered based on session metadata can be sent over to specific trafficmanaged queues that are arbitrated out to output ports using anarbitration circuit (not shown). The arbitration circuit can feed thesepacket flows to a downstream packet processing/terminating resourcedirectly. Certain embodiments provide for integration of thread andqueue management so as to enhance the throughput of downstream resourceshandling termination of network data through above said threads.

As will be understand, multiple types of conventional input/outputbusses such as PCI, Fibre Channel can be used in system embodimentsdescribed herein. The bus architecture can also be based on relevantJEDEC standards, on DIMM data transfer protocols, on Hypertransport, orany other suitable high speed, low latency interconnection system.Offload processor(s) 118 may include double data rate (DDR) dynamicrandom access memory (DRAM), reduced latency DRAM (RLDRAM), embeddedDRAM, next generation stacked memory such as Hybrid Memory Cube (HMC),flash, or other suitable memory, separate logic or bus management chips,programmable units such as field programmable gate arrays (FPGAs),custom designed application specific integrated circuits (ASICs) and anenergy efficient, general purpose processor such as those based on ARM,ARC, Tensilica, MIPS, Strong/ARM, or RISC architectures. Hostprocessor(s) 110 can include general purpose processor(s), includingthose based on Intel or AMD x86 architecture, Intel Itaniumarchitecture, MIPS architecture, SPARC architecture or the like.

FIG. 1-1 illustrates one embodiment of a hardware scheduled data flowmethod 140 suitable for operation in conjunction with an embodiment likethat of FIG. 1-0. As seen in flowchart 140, a hardware scheduler canmanage traffic by segregating packets based on sessions (141). In someembodiments, sessions are identified by metadata of the packets.Sessions can be prioritized and queued (142) and a general purposeoperating system (OS) running on one or more offload processors can beused control execution of a current session (143). A hardware schedulercan use the current state of the OS, including numbers of session, stateof the session, feedback from the OS relating to processing resources orfuture scheduling requirements, etc., to make scheduling decisions orarbitration between competing processes for memory resources (144). Ifcertain conditions are met, the hardware scheduler can initiate acontext switch in which a current session has its state stored inmemory, and a new session is begun or returned to (145).

FIG. 1-2 illustrates one embodiment of a hardware scheduler 150 (i.e., ascheduling circuit). The hardware scheduler 150 can include input ports152/152′, classification circuit 154, input queues (one shown as 156),scheduling circuits 158/158′, output queues (one shown as 160),arbitration circuit 162, and output ports 164/164′. Connected to thehardware scheduler can be common packet status registers 166/166′,packet buffers 168/168′, one or more cache memory ports 170 (which canbe an accelerator coherency port ACP, in one particular embodiment), anda low latency memory 172. It is understood that while FIG. 1-2 shows anarchitecture with two input ports and two output ports, alternateembodiments can include one input and output port, or more than two suchports.

The hardware scheduler 150 can receive packets from an arbiter circuit(not shown) that is connected to several such hardware schedulers. Thehardware scheduler 150 can receive such data at one or more input ports152/152′. The hardware scheduler 150 can employ a classification circuit154, which can examine incoming packets, and based on metadata presentin the packets, classifies packets into different incoming queues. Theclassification circuit 154 can examine different packet headers, and canuse an interval matching circuit to carry out segregation of incomingpackets. One suitable interval matching circuit is described in U.S.Pat. No. 7,760,715 issued to Dalal on Aug. 4, 2007 (hereinafter the '715patent). However, any other suitable classification scheme may beemployed to execute the classification circuit.

The hardware scheduler 150 can be connected with packet status registers166/166′ for communicating with offload processors (not shown).Registers 166/166′ can be operated upon by both the hardware scheduler150 and an OS running on an offload processor. The hardware schedulercan also be connected with a packet buffer 168/168′ to store outgoingpackets of a session or for processing to/by an offload processor OS. Adetailed explanation of packet status registers and packet buffers thatcan be included in embodiments is given below.

The hardware scheduler 150 can use port 170 to access data related to asession that is currently running on an offload processor OS in thecache of the offload processor and transfer it out using a bulk transferduring a context switch to a different session. The hardware scheduler150 can uses the cache transfer to reduce the overhead associated withthe session. The hardware scheduler 150 can also use a low latencymemory 172 to store the session related information from the cache forits subsequent access.

As noted above, a hardware scheduler 150 can have more than one inputport 152/152′. The data coming into the hardware scheduler may be packetdata waiting to be terminated at the offload processors or it could bepacket data waiting to be processed, modified or switched out. Thehardware scheduler 150 can be responsible for segregating incomingpackets into corresponding application sessions based on examination ofpacket data. The hardware scheduler 150 can be capable of packetinspection and identifying relevant packet characteristics.

A hardware scheduler 150 may offload part of the network stack to freeoffload processors from overhead incurred from such network stackprocessing. A hardware scheduler 150 may carry out any of TCP/transportoffload, encryption/decryption offload, segmentation and reassembly, orthe like, thus allowing the offload processor to use the payload of thenetwork packets directly.

In some embodiments, hardware scheduler 150 can further have thecapability to transfer the packets belonging to a session into aparticular traffic management queue (e.g., 156) for its scheduling (158)and transfer to an output queues (e.g., 160). The hardware scheduler 150may be used to control the scheduling of each of these persistentsessions into a general purpose OS. The stickiness of sessions across apipeline of stages, including a general purpose OS, can be accentuatedby a scheduler circuit 150 carrying out optimizations at each of thestages in the pipeline (described in more detail below). For example, ahardware scheduler 150 can takes into account of downstream executionresources. The session flows queued in each of these queues can be sentout through an output port to a downstream network element. Oneparticular implementation of such scheduling is shown in the '715patent, which is incorporated herein by reference, in its entirety.

Referring still to FIG. 1-2, a hardware scheduler 150 may employ anarbitration circuit 162 to arbitrate or otherwise control access ofmultiple traffic management output queues to available output ports.Each of the output ports may be connected to one of the offloadprocessor cores through a packet buffer 168/168′. A packet buffer168/168′ may further include a header pool and a packet body pool. Aheader pool can contain only the header of packets to be processed byoffload processors. Sometimes, if the size of the packet to be processedis sufficiently small, the header pool may contain the entire packet.Packets can be transferred over to the header pool/packet body pooldepending on the nature of operation carried out at the offloadprocessor. For packet processing, overlay, analytics, filtering and suchother applications it might be appropriate to transfer only the packetheader to the offload processors. In this case, depending on thehandling of the packet header, the packet body might either be sewntogether with a packet header and transferred over an egress interface,or dropped. For applications requiring the termination of packets, theentire body of the packet might be transferred. The offload processorcores may receive the packets and execute suitable application sessionon them to execute said packet contents.

A hardware scheduler 150 can schedule different sessions on a downstreamprocessor, wherein the two are operated in coordination to reduce theoverhead during context switches. A hardware scheduler 150 can beunderstood to arbitrate not just between outgoing queues or sessionflows at line rate speeds, but between terminated sessions at very highspeeds. The hardware scheduler 150 can manage the queuing of sessions onthe offload processor. A scheduling circuit 158/158′ can be responsiblefor queuing each session flow into the OS as a different OS processingentity. Scheduling circuit 158/158′ can be responsible for causing theexecution of a new application session on the OS. It can indicate to theOS that packets for a new session are available based on trafficmanagement carried out by it.

A hardware scheduler 150 can be informed of the state of the executionresources on the offload processors, the current session that is run onthe execution resource, the memory space allocated to it, and thelocation of the session context in the processor cache. It can use thestate of the execution resource to carry out traffic management andarbitration decisions. The hardware scheduler 150 can provide for anintegration of thread management on the operating system with trafficmanagement of incoming packets. It can induce persistence of sessionflows across a spectrum of components including traffic managementqueues and processing entities on the offload processors. An OS runningon a downstream (e.g. offload) processor may allocate executionresources such as processor cycles and memory to a particular queue itis currently handling. The OS may further allocate a thread or a groupof threads for that particular queue, so that it is handled distinctlyby the general purpose processing element as a separate entity. The factthat there are multiple sessions running on a general purpose (GP)processing resource, each handling data from a particular session flowresident in a queue on the hardware scheduler, can tightly integrate thehardware scheduler and the downstream resource. This can bring anelement of persistence within session information across the trafficmanagement and scheduling circuit and the general purpose processingresource. Further, the offload OS can be modified to reduce the penaltyand overhead associated with context switch between resources. This isfurther exploited by the hardware scheduler to carry out seamlessswitching between queues, and consequently their execution as differentsessions by the execution resource.

In effect, in some embodiments a hardware scheduler can be employed toimplement traffic management of incoming packets. Packets from a certainsource, relating to a certain traffic class, pertaining to a specificapplication or flowing to a certain socket are referred to as part of asession flow and can be classified using session metadata. Sessionmetadata often serve as the criterion by which packets are prioritizedand as such, incoming packets are reordered based on their sessionmetadata. This reordering of packets can occur in one or more buffersand can modify the traffic shape of these flows. Packets of a sessionthat are reordered based on session metadata can be sent over tospecific traffic managed queues that are arbitrated out to output portsusing an arbitration circuit. An arbitration circuit (e.g. 162) can feedthese packet flows to a downstream packet processing/terminatingresource directly. Certain embodiments provide for integration of threadand queue management so as to enhance the throughput of downstreamresources handling termination of network data through above saidthreads.

Accordingly, a hardware scheduler can perform any of the followingfunctions:

a) the hardware scheduler is responsible for carrying out trafficmanagement, arbitration and scheduling of incoming network packets (andflows);b) hardware scheduler is responsible for offloading part of the networkstack of the offload OS, so that the offload OS can be kept free ofstack level processing and resources are free to carry out execution ofapplication sessions;c) the hardware scheduler is responsible for classification of packetsbased on packet metadata, and packets classified into different sessionare queued in output traffic queues are sent over to the offload OS;d) the hardware scheduler is responsible for cooperating with minimaloverhead context switching between terminated sessions on the offloadOS; the hardware scheduler ensures that multiple sessions on the offloadOS can be switched with as minimal overhead as possible (the ability toswitch between multiple sessions on the offload sessions makes itpossible to terminate multiple sessions at very high speeds, providingpacket processing speeds for terminated sessions);d) the hardware scheduler is responsible for queuing each session flowinto the OS as a different OS processing entity;e) the hardware scheduler is responsible for causing the execution of anew application session on the OS, it can indicate to the OS thatpackets for a new session are available based on traffic managementcarried out by it;f) the hardware scheduler is informed of the state of the executionresources on the offload processors, the current session that is run onthe execution resource and the memory space allocated to it, thelocation of the session context in the processor cache. The hardwarescheduler can use the state of the execution resource to carry outtraffic management and arbitration decisions. The hardware scheduler canprovide for an integration of thread management on the operating systemwith traffic management of incoming packets. It can induce persistenceof session flows across a spectrum of components including trafficmanagement queues and processing entities on the offload processors.

As will be understood, many of the foregoing processing tasks can beimplemented on multiple threads running on multiple processing cores.Such parallelization of tasks into multiple thread contexts can providefor increased throughput. Processors architectures such as MIPS mayinclude deep instruction pipelines to improve the number of instructionsper cycle. Further, the ability to run a multi-threaded programmingenvironment results in enhanced usage of existing processor resources.To further increase parallel execution on the hardware, processorarchitecture may include multiple processor cores. Multi-corearchitectures comprising the same type of cores, referred to ashomogeneous core architectures, provide higher instruction throughput byparallelizing threads or processes across multiple cores. However, insuch homogeneous core architectures, the shared resources, such asmemory, are amortized over a small number of processors. In still otherembodiments, multiple offload or host processors can reside on modulesconnected to individual rack units or blades that in turn reside onracks or individual servers. These can be further grouped into clustersand datacenters, which can be spatially located in the same building, inthe same city, or even in different countries. Any grouping level can beconnected to each other, and/or connected to public or private cloudinternets.

Memory and I/O accesses can incur a high amount of processor overhead.Further, context switches in conventional general purpose processingunits can be computationally intensive. It is therefore desirable toreduce context switch overhead in a networked computing resourcehandling a plurality of networked applications in order to increaseprocessor throughput. Conventional server loads can require complextransport, high memory bandwidth, extreme amounts of data bandwidth(randomly accessed, parallelized, and highly available), but often withlight touch processing: HTML, video, packet-level services, security,and analytics. Further, idle processors still consume more than 50% oftheir peak power consumption.

In contrast, according to embodiments herein, complex transport, databandwidth intensive, frequent random access oriented, ‘light’ touchprocessing loads can be handled behind a socket abstraction created onmultiple offload processor cores. At the same time, “heavy” touch,computing intensive loads can be handled by a socket abstraction on ahost processor core (e.g., x86 processor cores). Such software socketscan allow for a natural partitioning of these loads between ARM and x86processor cores. By usage of new application level sockets, according toembodiments, server loads can be broken up across the offload processingcores and the host processing cores.

FIGS. 2-0 to 2-5 describe aspects of hardware embodiments and methodsfor providing scheduling and traffic management services usingprocessing modules. In particular embodiments, such processing modulescan include DIMM mountable modules to support offload processing.

FIG. 2-0 is a block diagram of a processing module 200 according to oneembodiment. A processing module 200 can include a physical connector202, a memory interface 204, arbiter logic 206, offload processor(s)208, local memory 210, and control logic 212. A connector 202 canprovide a physical connection to system memory bus. This is in contrastto a host processor which can access a system memory bus via a memorycontroller, or the like. In very particular embodiments, a connector 202can be compatible with a dual in-line memory module (DIMM) slot of acomputing system. Accordingly, a system including multiple DIMM slotscan be populated with one or more processing modules 200, or a mix ofprocessing modules and DIMM modules.

A memory interface 204 can detect data transfers on a system memory bus,and in appropriate cases, enable write data to be stored in theprocessing module 200 and/or read data to be read out from theprocessing module 200. Such data transfers can include the receipt ofpacket data having a particular network identifier. In some embodiments,a memory interface 204 can be a slave interface, thus data transfers arecontrolled by a master device separate from the processing module 200.In very particular embodiments, a memory interface 204 can be a directmemory access (DMA) slave, to accommodate DMA transfers over a systemmemory bus initiated by a DMA master. In some embodiments, a DMA mastercan be a device different from a host processor. In such configurations,processing module 200 can receive data for processing (e.g., DMA write),and transfer processed data out (e.g., DMA read) without consuming hostprocessor resources.

A memory interface 204 can detect data transfers on a system memory bus,and in appropriate cases, enable write data to be stored in theprocessing module 200 and/or read data to be read out from theprocessing module 200. In some embodiments, a memory interface 204 canbe a slave interface, thus data transfers are controlled by a masterdevice separate from the processing module. In very particularembodiments, a memory interface 204 can be a direct memory access (DMA)slave, to accommodate DMA transfers over a system memory initiated by aDMA master. In some embodiments, a DMA master can be a device differentfrom a host processor. In such configurations, processing module 200 canreceive data for processing (e.g., DMA write), and transfer processeddata out (e.g., DMA read) without consuming host processor resources.

Arbiter logic 206 can arbitrate between conflicting accesses of datawithin processing module 200. In some embodiments, arbiter logic 206 canarbitrate between accesses by offload processor 208 and accessesexternal to the processor module 200. It is understood that a processingmodule 200 can include multiple locations that are operated on at thesame time. It is understood that accesses arbitrated by arbiter logic206 can include accesses to physical system memory space occupied by theprocessor module 200, as well as accesses to other resources (e.g.,cache memory of offload or host processor). Accordingly, arbitrationrules for arbiter logic 206 can vary according to application. In someembodiments, such arbitration rules are fixed for a given processormodule 200. In such cases, different applications can be accommodated byswitching out different processing modules. However, in alternateembodiments, such arbitration rules can be configurable.

Offload processor 208 can include one or more processors that canoperate on data transferred over the system memory bus. In someembodiments, offload processors can run a general operating system orserver applications such as Apache (as but one very particular example),enabling processor contexts to be saved and retrieved. Computing tasksexecuted by offload processor 208 can be handled by the hardwarescheduler. Offload processors 208 can operate on data buffered in theprocessor module 200. In addition or alternatively, offload processors208 can access data stored elsewhere in a system memory space. In someembodiments, offload processors 208 can include a cache memoryconfigured to store context information. An offload processor 208 caninclude multiple cores or one core.

A processor module 200 can be included in a system having a hostprocessor (not shown). In some embodiments, offload processors 208 canbe a different type of processor as compared to the host processor. Inparticular embodiments, offload processors 208 can consume less powerand/or have less computing power than a host processor. In veryparticular embodiments, offload processors 208 can be “wimpy” coreprocessors, while a host processor can be a “brawny” core processor.However, in alternate embodiments, offload processors 208 can haveequivalent computing power to any host processor. In very particularembodiments, a host processor can be an x86 type processor, while anoffload processor 208 can include an ARM, ARC, Tensilica, MIPS,Strong/ARM, or RISC type processor, as but a few examples.

Local memory 210 can be connected to offload processor 208 to enable thestoring of context information. Accordingly, an offload processor 208can store current context information, and then switch to a newcomputing task, then subsequently retrieve the context information toresume the prior task. In very particular embodiments, local memory 210can be a low latency memory with respect to other memories in a system.In some embodiments, storing of context information can include copyingan offload processor 208 cache.

In some embodiments, a same space within local memory 210 is accessibleby multiple offload processors 208 of the same type. In this way, acontext stored by one offload processor can be resumed by a differentoffload processor.

Control logic 212 can control processing tasks executed by offloadprocessor(s). In some embodiments, control logic 212 can be considered ahardware scheduler that can be conceptualized as including a dataevaluator 214, scheduler 216 and a switch controller 218. A dataevaluator 214 can extract “metadata” from write data transferred over asystem memory bus. “Metadata”, as used herein, can be any informationembedded at one or more predetermined locations of a block of write datathat indicates processing to be performed on all or a portion of theblock of write data and/or indicate a particular task/process to whichthe data belongs (e.g., classification data). In some embodiments,metadata can be data that indicates a higher level organization for theblock of write data. As but one very particular embodiment, metadata canbe header information of one or more network packets (which may or maynot be encapsulated within a higher layer packet structure).

A scheduler 216 (e.g., a hardware scheduler) can order computing tasksfor offload processor(s) 208. In some embodiments, scheduler 216 cangenerate a schedule that is continually updated as write data forprocessing is received. In very particular embodiments, a scheduler 216can generate such a schedule based on the ability to switch contexts ofoffload processor(s) 208. In this way, on-module computing prioritiescan be adjusted on the fly. In very particular embodiments, a scheduler216 can assign a portion of physical address space (e.g., memorylocations within local memory 210) to an offload processor 208,according to computing tasks. The offload processor 208 can then switchbetween such different spaces, saving context information prior to eachswitch, and subsequently restoring context information when returning tothe memory space.

Switch controller 218 can control computing operations of offloadprocessor(s) 208. In particular embodiments, according to scheduler 216,switch controller 218 can order offload processor(s) 208 to switchcontexts. It is understood that a context switch operation can be an“atomic” operation, executed in response to a single command from switchcontroller 218. In addition or alternatively, a switch controller 218can issue an instruction set that stores current context information,recalls context information, etc.

In some embodiments, processor module 200 can include a buffer memory(not shown). A buffer memory can store received write data on board theprocessor module. A buffer memory can be implemented on an entirelydifferent set of memory devices, or can be a memory embedded with logicand/or the offload processor. In the latter case, arbiter logic 206 canarbitrate access to the buffer memory. In some embodiments, a buffermemory can correspond to a portion of a system physical memory space.The remaining portion of the system memory space can correspond to otherlike processor modules and/or memory modules connected to the samesystem memory bus. In some embodiments buffer memory can be differentthan local memory 210. For example, buffer memory can have a sloweraccess time than local memory 210. However, in other embodiments, buffermemory and local memory can be implemented with like memory devices.

In very particular embodiments, write data for processing can have anexpected maximum flow rate. A processor module 200 can be configured tooperate on such data at, or faster than, such a flow rate. In this way,a master device (not shown) can write data to a processor module withoutdanger of overwriting data “in process”.

The various computing elements of a processor module 200 can beimplemented as one or more integrated circuit devices (ICs). It isunderstood that the various components shown in FIG. 2-0 can be formedin the same or different ICs. For example, control logic 212, memoryinterface 214, and/or arbiter logic 206 can be implemented on one ormore logic ICs, while offload processor(s) 208 and local memory 210 areseparate ICs. Logic ICs can be fixed logic (e.g., application specificICs), programmable logic (e.g., field programmable gate arrays, FPGAs),or combinations thereof.

Advantageously, the foregoing hardware and systems can provide improvedcomputational performance as compared to traditional computing systems.Conventional systems, including those based on x86 processors, are oftenill-equipped to handle such high volume applications. Even idling, x86processors use a significant amount of power, and near continuousoperation for high bandwidth packet analysis or other high volumeprocessing tasks makes the processor energy costs one of the dominantprice factors.

In addition, conventional systems can have issues with the high cost ofcontext switching wherein a host processor is required to executeinstructions which can include switching from one thread to another.Such a switch can require storing and recalling the context for thethread. If such context data is resident in a host cache memory, such acontext switch can occur relatively quickly. However, if such contextdata is no longer in cache memory (i.e., a cache miss), the data must berecalled from system memory, which can incur a multi-cycle latency.Continuous cache misses during context switching can adversely impactsystem performance.

FIG. 2-1 shows a processor module 200-1 according to one very particularembodiment which is capable of reducing issues associated with highvolume processing or context switching associated with many conventionalserver systems. A processor module 200-1 can include ICs 220-0/1 mountedto a printed circuit board (PCB) type substrate 222. PCB type substrate222 can include in-line module connector 202, which in one veryparticular embodiment, can be a DIMM compatible connector. IC 220-0 canbe a system-on-chip (SoC) type device, integrating multiple functions.In the very particular embodiment shown, an IC 220-0 can includeembedded processor(s), logic and memory. Such embedded processor(s) canbe offload processor(s) 208 as described herein, or equivalents. Suchlogic can be any of controller logic 212, memory interface 204 and/orarbiter logic 206, as described herein, or equivalents. Such memory canbe any of local memory 210, cache memory for offload processor(s) 208,or buffer memory, as described herein, or equivalents. Logic IC 220-1can provide logic functions not included IC 220-0.

FIG. 2-2 shows a processor module 200-2 according to another veryparticular embodiment. A processor module 200-2 can include ICs 220-2,-3, -4, -5 mounted to a PCB type substrate 222, like that of FIG. 2-1.However, unlike FIG. 2-1, processor module functions are distributedamong single purpose type ICs. IC 220-2 can be a processor IC, which canbe an offload processor 208. IC 220-3 can be a memory IC which caninclude local memory 210, buffer memory, or combinations thereof. IC220-4 can be a logic IC which can include control logic 212, and in onevery particular embodiment, can be an FPGA. IC 220-5 can be anotherlogic IC which can include memory interface 204 and arbiter logic 206,and in one very particular embodiment, can also be an FPGA.

It is understood that FIGS. 2-1/2 represent but two of variousimplementations. The various functions of a processor module can bedistributed over any suitable number of ICs, including a single SoC typeIC.

FIG. 2-3 shows an opposing side of a processor module 200-1 or 200-2according to a very particular embodiment. Processor module 200-3 caninclude a number of memory ICs, one shown as 220-6, mounted to a PCBtype substrate 222, like that of FIG. 2-1. It is understood that variousprocessing and logic components can be mounted on an opposing side tothat shown. A memory IC 220-6 can be configured to represent a portionof the physical memory space of a system. Memory ICs 220-6 can performany or all of the following functions: operate independently of otherprocessor module components, providing system memory accessed in aconventional fashion; serve as buffer memory, storing write data thatcan be processed with other processor module components, or serve aslocal memory for storing processor context information.

FIG. 2-4 shows a conventional DIMM module (i.e., it serves only a memoryfunction) that can populate a memory bus along with processor modules asdescribed herein, or equivalents.

FIG. 2-5 shows a system 230 according to one embodiment. A system 230can include a system memory bus 228 accessible via multiple in-linemodule slots (one shown as 226). According to embodiments, any or all ofthe slots 226 can be occupied by a processor module 200 as describedherein, or an equivalent. In the event all slots 226 are not occupied bya processor module 200, available slots can be occupied by conventionalin-line memory modules 224. In a very particular embodiment, slots 226can be DIMM slots.

In some embodiments, a processor module 200 can occupy one slot.However, in other embodiments, a processor module can occupy multipleslots.

In some embodiments, a system memory bus 228 can be further interfacedwith one or more host processors and/or input/output device (not shown).

Having described processor modules according to various embodiments,operations of an offload processor module capable of interfacing withserver or similar system via a memory bus and according to a particularembodiment will now be described.

FIG. 3 shows a system 301 according to another embodiment. A system 301can transport packet data requiring network overlay services to one ormore computational units (one shown as 300) located on a module, whichin particular embodiments, can include a connector compatible with anexisting memory module. In some embodiments, a computational unit 300can include a processor module as described in embodiments herein, or anequivalent. A computational unit 300 can be capable of intercepting orotherwise accessing packets sent over a memory bus 316 and carrying outprocessing on such packets, including but not limited to termination ormetadata processing. A system memory bus 316 can be a system memory buslike those described herein, or equivalents (e.g., 228).

Referring still to FIG. 3, a system 301 can include an I/O device 302which can receive packet or other I/O data from an external source. Insome embodiments I/O device 302 can include physical or virtualfunctions generated by the physical device to receive a packet or otherI/O data from the network or another computer or virtual machine. In thevery particular embodiment shown, an I/O device 302 can include anetwork interface card (NIC) having input buffer 302 a (e.g., DMA ringbuffer) and an I/O virtualization function 302 b.

According to embodiments, an I/O device 302 can write a descriptorincluding details of the necessary memory operation for the packet (i.e.read/write, source/destination). Such a descriptor can be assigned avirtual memory location (e.g., by an operating system of the system301). I/O device 302 then communicates with an input output memorymanagement unit (IOMMU) 304 which can translate virtual addresses tocorresponding physical addresses with an IOMMU function 304 b. In theparticular embodiment shown, a translation look-aside buffer (TLB) 304 acan be used for such translation. Virtual function reads or writes databetween I/O device and system memory locations can then be executed witha direct memory transfer (e.g., DMA) via a memory controller 306 b ofthe system 301. An I/O device 302 can be connected to IOMMU 304 by ahost bus 312. In one very particular embodiment, a host bus 312 can be aperipheral interconnect (PCI) type bus. IOMMU 304 can be connected to ahost processing section 306 at a central processing unit I/O (CPUIO) 306a. In the embodiment shown, such a connection 314 can support aHyperTransport (HT) protocol.

In the embodiment shown, a host processing section 306 can include theCPUIO 306 a, memory controller 306 b, processing core 306 c andcorresponding provisioning agent 306 d.

In particular embodiments, a computational unit 300 can interface withthe system bus 316 via standard in-line module connection, which in veryparticular embodiments can include a DIMM type slot. In the embodimentshown, a memory bus 316 can be a DDR3 type memory bus. Alternateembodiments can include any suitable system memory bus. Packet data canbe sent by memory controller 306 b via memory bus 316 to a DMA slaveinterface 310 a. DMA slave interface 310 a can be adapted to receiveencapsulated read/write instructions from a DMA write over the memorybus 316.

A hardware scheduler (308 b/c/d/e/h) can perform traffic management onincoming packets by categorizing them according to flow using sessionmetadata. Packets can be queued for output in an onboard memory (310b/308 a/308 m) based on session priority. When the hardware schedulerdetermines that a packet for a particular session is ready to beprocessed by the offload processor 308 i, the onboard memory is signaledfor a context switch to that session. Utilizing this method ofprioritization, context switching overhead can be reduced, as comparedto conventional approaches. That is, a hardware scheduler can handlecontext switching decisions and thus optimize the performance of thedownstream resource (e.g., offload processor 308 i).

As noted above, in very particular embodiments, an offload processor 308i can be a “wimpy core” type processor. According to some embodiments, ahost processor 306 c can be a “brawny core” type processor (e.g., an x86or any other processor capable of handling “heavy touch” computationaloperations). While an I/O device 302 can be configured to trigger hostprocessor interrupts in response to incoming packets, according toembodiments, such interrupts can be disabled, thereby reducingprocessing overhead for the host processor 306 c. In some veryparticular embodiments, an offload processor 308 i can include an ARM,ARC, Tensilica, MIPS, Strong/ARM or any other processor capable ofhandling “light touch” operations. Preferably, an offload processor canrun a general purpose operating system for executing a plurality ofsessions, which can be optimized to work in conjunction with thehardware scheduler in order to reduce context switching overhead.

Referring still to FIG. 3, in operation, a system 301 can receivepackets from an external network over a network interface. The packetsare destined for either a host processor 306 c or an offload processor308 i based on the classification logic and schematics employed by I/Odevice 302. In particular embodiments, I/O device 302 can operate as avirtualized NIC, with packets for a particular logical network or to acertain virtual MAC (VMAC) address can be directed into separate queuesand sent over to the destination logical entity. Such an arrangement cantransfer packets to different entities. In some embodiments, each suchentity can have a virtual driver, a virtual device model that it uses tocommunicate with connected virtual network.

According to embodiments, multiple devices can be used to redirecttraffic to specific memory addresses. So, each of the network devicesoperates as if it is transferring the packets to the memory location ofa logical entity. However, in reality, such packets are transferred tomemory addresses where they can be handled by one or more offloadprocessors (e.g., 308 i). In particular embodiments such transfers areto physical memory addresses, thus logical entities can be removed fromthe processing, and a host processor can be free from such packethandling.

Accordingly, embodiments can be conceptualized as providing a memory“black box” to which specific network data can be fed. Such a memoryblack box can handle the data (e.g., process it) and respond back whensuch data is requested.

Referring still to FIG. 3, according to some embodiments, I/O device 302can receive data packets from a network or from a computing device. Thedata packets can have certain characteristics, including transportprotocol number, source and destination port numbers, source anddestination IP addresses, for example. The data packets can further havemetadata that is processed (308 d) that helps in their classificationand management.

I/O device 302 can include, but is not limited to, peripheral componentinterconnect (PCI) and/or PCI express (PCIe) devices connecting with ahost motherboard via PCI or PCIe bus (e.g., 312). Examples of I/Odevices include a network interface controller (NIC), a host busadapter, a converged network adapter, an ATM network interface, etc.

In order to provide for an abstraction scheme that allows multiplelogical entities to access the same I/O device 302, the I/O device maybe virtualized to provide for multiple virtual devices each of which canperform some of the functions of the physical I/O device. The IOvirtualization program (e.g., 302 b) according to an embodiment, canredirect traffic to different memory locations (and thus to differentoffload processors attached to modules on a memory bus). To achievethis, an I/O device 302 (e.g., a network card) may be partitioned intoseveral function parts; including controlling function (CF) supportinginput/output virtualization (IOV) architecture (e.g., single-root IOV)and multiple virtual function (VF) interfaces. Each virtual functioninterface may be provided with resources during runtime for dedicatedusage. Examples of the CF and VF may include the physical function andvirtual functions under schemes such as Single Root I/O Virtualizationor Multi-Root I/O Virtualization architecture. The CF acts as thephysical resources that sets up and manages virtual resources. The CF isalso capable of acting as a full-fledged IO device. The VF isresponsible for providing an abstraction of a virtual device forcommunication with multiple logical entities/multiple memory regions.

The operating system/the hypervisor/any of the virtual machines/usercode running on a host processor 306 c may be loaded with a devicemodel, a VF driver and a driver for a CF. The device model may be usedto create an emulation of a physical device for the host processor 306 cto recognize each of the multiple VFs that are created. The device modelmay be replicated multiple times to give the impression to a VF driver(a driver that interacts with a virtual IO device) that it isinteracting with a physical device of a particular type.

For example, a certain device module may be used to emulate a networkadapter such as the Intel® Ethernet Converged Network Adapter(CNA)X540-T2, so that the I/O device 302 believes it is interacting with suchan adapter. In such a case, each of the virtual functions may have thecapability to support the functions of the above said CNA, i.e., each ofthe Physical Functions should be able to support such functionality. Thedevice model and the VF driver can be run in either privileged ornon-privileged mode. In some embodiments, there is no restriction withregard to who hosts/runs the code corresponding to the device model andthe VF driver. The code, however, has the capability to create multiplecopies of device model and VF driver so as to enable multiple copies ofsaid I/O interface to be created.

An application or provisioning agent 306 d, as part of anapplication/user level code running in a kernel, may create a virtualI/O address space for each VF, during runtime and allocate part of thephysical address space to it. For example, if an application handlingthe VF driver instructs it to read or write packets from or to memoryaddresses 0xaaaa to 0xffff, the device driver may write I/O descriptorsinto a descriptor queue with a head and tail pointer that are changeddynamically as queue entries are filled. The data structure may be ofanother type as well, including but not limited to a ring structure 302a or hash table.

The VF can read from or write data to the address location pointed to bythe driver. Further, on completing the transfer of data to the addressspace allocated to the driver, interrupts, which are usually triggeredto the host processor to handle said network packets, can be disabled.Allocating a specific I/O space to a device can include allocating saidIO space a specific physical memory space occupied.

In another embodiment, the descriptor may comprise only a writeoperation, if the descriptor is associated with a specific datastructure for handling incoming packets. Further, the descriptor foreach of the entries in the incoming data structure may be constant so asto redirect all data write to a specific memory location. In analternate embodiment, the descriptor for consecutive entries may pointto consecutive entries in memory so as to direct incoming packets toconsecutive memory locations.

Alternatively, said operating system may create a defined physicaladdress space for an application supporting the VF drivers and allocatea virtual memory address space to the application or provisioning agent306 d, thereby creating a mapping for each virtual function between saidvirtual address and a physical address space. Said mapping betweenvirtual memory address space and physical memory space may be stored inIOMMU tables (e.g., a TLB 304 a). The application performing memoryreads or writes may supply virtual addresses to say virtual function,and the host processor OS may allocate a specific part of the physicalmemory location to such an application.

Alternatively, VF may be configured to generate requests such as readand write which may be part of a direct memory access (DMA) read orwrite operation, for example. The virtual addresses is be translated bythe IOMMU 304 to their corresponding physical addresses and the physicaladdresses may be provided to the memory controller for access. That is,the IOMMU 304 may modify the memory requests sourced by the I/O devicesto change the virtual address in the request to a physical address, andthe memory request may be forwarded to the memory controller for memoryaccess. The memory request may be forwarded over a bus 314 that supportsa protocol such as HyperTransport 314. The VF may in such cases carryout a direct memory access by supplying the virtual memory address tothe IOMMU 304.

Alternatively, said application may directly code the physical addressinto the VF descriptors if the VF allows for it. If the VF cannotsupport physical addresses of the form used by the host processor 306 c,an aperture with a hardware size supported by the VF device may be codedinto the descriptor so that the VF is informed of the target hardwareaddress of the device. Data that is transferred to an aperture may bemapped by a translation table to a defined physical address space in thesystem memory. The DMA operations may be initiated by software executedby the processors, programming the I/O devices directly or indirectly toperform the DMA operations.

Referring still to FIG. 3, in particular embodiments, parts ofcomputational unit 300 can be implemented with one or more FPGAs. In thesystem of FIG. 3, computational unit 300 can include FPGA 310 in whichcan be formed a DMA slave device module 310 a and arbiter 310 f. A DMAslave module 310 a can be any device suitable for attachment to a memorybus 316 that can respond to DMA read/write requests. In alternateembodiments, a DMA slave module 310 a can be another interface capableof block data transfers over memory bus 316. The DMA slave module 310 acan be capable of receiving data from a DMA controller (when it performsa read from a ‘memory’ or from a peripheral) or transferring data to aDMA controller (when it performs a write instruction on the DMA slavemodule 310 a). The DMA slave module 310 a may be adapted to receive DMAread and write instructions encapsulated over a memory bus, (e.g., inthe form of a DDR data transmission, such as a packet or data burst), orany other format that can be sent over the corresponding memory bus.

A DMA slave module 310 a can reconstruct the DMA read/write instructionfrom the memory R/W packet. The DMA slave module 310 a may be adapted torespond to these instructions in the form of data reads/data writes tothe DMA master, which could either be housed in a peripheral device, inthe case of a PCIe bus, or a system DMA controller in the case of an ISAbus.

I/O data that is received by the DMA device 310 a can then queued forarbitration. Arbitration can include the process of scheduling packetsof different flows, such that they are provided access to availablebandwidth based on a number of parameters. In general, an arbiter 310 fprovides resource access to one or more requestors. If multiplerequestors request access, an arbiter 310 f can determine whichrequestor becomes the accessor and then passes data from the accessor tothe resource interface, and the downstream resource can begin executionon the data. After the data has been completely transferred to aresource, and the resource has competed execution, the arbiter 310 f cantransfer control to a different requestor and this cycle repeats for allavailable requestors. In the embodiment of FIG. 3 arbiter 310 f cannotify other portions of computational unit 300 (e.g., 308) of incomingdata.

Alternatively, a computation unit 300 can utilize an arbitration schemeshown in U.S. Pat. No. 7,813,283, issued to Dalal on Oct. 12, 2010, thecontents of which are incorporated herein by reference. Other suitablearbitration schemes known in art could be implemented in embodimentsherein. Alternatively, the arbitration scheme of the current inventionmight be implemented using an OpenFlow switch and an OpenFlowcontroller.

In the very particular embodiment of FIG. 3, computational unit 300 canfurther include notify/prefetch circuits 310 c which can prefetch datastored in a buffer memory 310 b in response to DMA slave module 310 a,and as arbitrated by arbiter 310 f. Further, arbiter 310 f can accessother portions of the computational unit 300 via a memory mapped I/Oingress path 310 e and egress path 310 g.

Referring to FIG. 3, a hardware scheduler can include a schedulingcircuit 308 b/n to implement traffic management of incoming packets.Packets from a certain source, relating to a certain traffic class,pertaining to a specific application or flowing to a certain socket arereferred to as part of a session flow and are classified using sessionmetadata. Such classification can be performed by classifier 308 e.

In some embodiments, session metadata 308 d can serve as the criterionby which packets are prioritized and scheduled and as such, incomingpackets can be reordered based on their session metadata. Thisreordering of packets can occur in one or more buffers and can modifythe traffic shape of these flows. The scheduling discipline chosen forthis prioritization, or traffic management (TM), can affect the trafficshape of flows and micro-flows through delay (buffering), bursting oftraffic (buffering and bursting), smoothing of traffic (buffering andrate-limiting flows), dropping traffic (choosing data to discard so asto avoid exhausting the buffer), delay jitter (temporally shifting cellsof a flow by different amounts) and by not admitting a connection (e.g.,cannot simultaneously guarantee existing service level agreements (SLAs)with an additional flow's SLA).

According to embodiments, computational unit 300 can serve as part of aswitch fabric, and provide traffic management with depth-limited outputqueues, the access to which is arbitrated by a scheduling circuit 308b/n. Such output queues are managed using a scheduling discipline toprovide traffic management for incoming flows. The session flows queuedin each of these queues can be sent out through an output port to adownstream network element.

It is noted that conventional traffic management do not take intoaccount the handling and management of data by downstream elementsexcept for meeting the SLA agreements it already has with saiddownstream elements.

In contrast, according to embodiments a scheduler circuit 308 b/n canallocate a priority to each of the output queues and carry outreordering of incoming packets to maintain persistence of session flowsin these queues. A scheduler circuit 308 b/n can be used to control thescheduling of each of these persistent sessions into a general purposeoperating system (OS) 308 j, executed on an offload processor 308 i.Packets of a particular session flow, as defined above, can belong to aparticular queue. The scheduler circuit 308 b/n may control theprioritization of these queues such that they are arbitrated forhandling by a general purpose (GP) processing resource (e.g., offloadprocessor 308 i) located downstream. An OS 308 j running on a downstreamprocessor 308 i can allocate execution resources such as processorcycles and memory to a particular queue it is currently handling. The OS308 j may further allocate a thread or a group of threads for thatparticular queue, so that it is handled distinctly by the generalpurpose processing element 308 i as a separate entity. The fact thatthere can be multiple sessions running on a GP processing resource, eachhandling data from a particular session flow resident in a queueestablished by the scheduler circuit, tightly integrates the schedulerand the downstream resource (e.g., 308 i). This can bring aboutpersistence of session information across the traffic management andscheduling circuit and the general purpose processing resource 308 i.

Dedicated computing resources (e.g., 308 i), memory space and sessioncontext information for each of the sessions can provide a way ofhandling, processing and/or terminating each of the session flows at thegeneral purpose processor 308 i. The scheduler circuit 308 b/n canexploit this functionality of the execution resource to queue sessionflows for scheduling downstream. The scheduler circuit 308 b/n can beinformed of the state of the execution resource(s) (e.g., 308 i), thecurrent session that is run on the execution resource; the memory spaceallocated to it, the location of the session context in the processorcache.

According to embodiments, a scheduler circuit 308 b/n can furtherinclude switching circuits to change execution resources from one stateto another. The scheduler circuit 308 b/n can use such a capability toarbitrate between the queues that are ready to be switched into thedownstream execution resource. Further, the downstream executionresource can be optimized to reduce the penalty and overhead associatedwith context switch between resources. This is further exploited by thescheduler circuit 308 b/n to carry out seamless switching betweenqueues, and consequently their execution as different sessions by theexecution resource.

According to embodiments, a scheduler circuit 308 b/n can scheduledifferent sessions on a downstream processing resource, wherein the twoare operated in coordination to reduce the overhead during contextswitches. An important factor in decreasing the latency of services andengineering computational availability can be hardware context switchingsynchronized with network queuing. In embodiments, when a queue isselected by a traffic manager, a pipeline coordinates swapping in of thecache (e.g., L2 cache) of the corresponding resource (e.g., 308 i) andtransfers the reassembled I/O data into the memory space of theexecuting process. In certain cases, no packets are pending in thequeue, but computation is still pending to service previous packets.Once this process makes a memory reference outside of the data swapped,the scheduler circuit (308 b/n) can enable queued data from an I/Odevice 302 to continue scheduling the thread.

In some embodiments, to provide fair queuing to a process not havingdata, a maximum context size can be assumed as data processed. In thisway, a queue can be provisioned as the greater of computational resourceand network bandwidth resource. As but one very particular example, acomputation resource can be an ARM A9 processor running at 800 MHz,while a network bandwidth can be 3 Gbps of bandwidth. Given the lopsidednature of this ratio, embodiments can utilize computation having manyparallel sessions (such that the hardware's prefetching ofsession-specific data offloads a large portion of the host processorload) and having minimal general purpose processing of data.

Accordingly, in some embodiments, a scheduler circuit 308 b/n can beconceptualized as arbitrating, not between outgoing queues at line ratespeeds, but arbitrating between terminated sessions at very high speeds.The stickiness of sessions across a pipeline of stages, including ageneral purpose OS, can be a scheduler circuit optimizing any or allsuch stages of such a pipeline.

Alternatively, a scheduling scheme can be used as shown in U.S. Pat. No.7,760,715 issued to Dalal on Jul. 20, 2010, incorporated herein byreference. This scheme can be useful when it is desirable to rate limitthe flows for preventing the downstream congestion of another resourcespecific to the over-selected flow, or for enforcing service contractsfor particular flows. Embodiments can include arbitration scheme thatallows for service contracts of downstream resources, such as generalpurpose OS that can be enforced seamlessly.

Referring still to FIG. 3, a hardware scheduler according to embodimentsherein, or equivalents, can provide for the classification of incomingpacket data into session flows based on session metadata. It can furtherprovide for traffic management of these flows before they are arbitratedand queued as distinct processing entities on the offload processors.

In some embodiments, offload processors (e.g., 308 i) can be generalpurpose processing units capable of handling packets of differentapplication or transport sessions. Such offload processors can be lowpower processors capable of executing general purpose instructions. Theoffload processors could be any suitable processor, including but notlimited to: ARM, ARC, Tensilica, MIPS, StrongARM or any other processorthat serves the functions described herein. Such offload processors havea general purpose OS running on them, wherein the general purpose OS isoptimized to reduce the penalty associated with context switchingbetween different threads or group of threads.

In contrast, context switches on host processors can be computationallyintensive processes that require the register save area, process contextin the cache and TLB entries to be restored if they are invalidated oroverwritten. Instruction Cache misses in host processing systems canlead to pipeline stalls and data cache misses lead to operation stalland such cache misses reduce processor efficiency and increase processoroverhead.

In contrast, an OS 308 j running on the offload processors 308 i inassociation with a scheduler circuit 308 b/n, can operate together toreduce the context switch overhead incurred between different processingentities running on it. Embodiments can include a cooperative mechanismbetween a scheduler circuit and the OS on the offload processor 308 i,wherein the OS sets up session context to be physically contiguous(physically colored allocator for session heap and stack) in the cache;then communicates the session color, size, and starting physical addressto the scheduler circuit upon session initialization. During an actualcontext switch, a scheduler circuit can identify the session context inthe cache by using these parameters and initiate a bulk transfer ofthese contents to an external low latency memory (e.g., 308 g). Inaddition, the scheduler circuit can manage the prefetch of the oldsession if its context was saved to a local memory 308 g. In particularembodiments, a local memory 308 g can be low latency memory, such as areduced latency dynamic random access memory (RLDRAM), as but one veryparticular embodiment. Thus, in embodiments, session context can beidentified distinctly in the cache.

In some embodiments, context size can be limited to ensure fastswitching speeds. In addition or alternatively, embodiments can includea bulk transfer mechanism to transfer out session context to a localmemory 308 g. The cache contents stored therein can then be retrievedand prefetched during context switch back to a previous session.Different context session data can be tagged and/or identified withinthe local memory 308 g for fast retrieval. As noted above, contextstored by one offload processor may be recalled by a different offloadprocessor.

In the very particular embodiment of FIG. 3, multiple offload processingcores can be integrated into a computation FPGA 308. Multiplecomputational FPGAs can be arbitrated by arbitrator circuits in anotherFPGA 310. The combination of computational FPGAs (e.g., 308) and arbiterFPGAs (e.g., 310) are referred to as “XIMM” modules or “Xockets DIMMmodules” (e.g., computation unit 300). In particular applications, theseXIMM modules can provide integrated traffic and thread managementcircuits that broker execution of multiple sessions on the offloadprocessors.

FIG. 3 also shows an offload processor tunnel connection 308 k, as wellas a memory interface 308 m and port 3081 (which can be an acceleratorcoherency port (ACP)). Memory interface 308 m can access buffer memory308 a.

Having described various embodiments suitable for hardware schedulingand traffic management operations, an example illustrating particularaspects will now be described.

FIG. 4 illustrates an example embodiment of a scheduling process 400 foraccess to offload processing resources according to a very particularembodiment. In some embodiments, a scheduler (e.g., hardware scheduler)can implement a scheduling process as a traffic management scheme inorder to meet the requirements of offload processors, and may beoperated in a preemption mode. In a preemption mode, the scheduler canbe responsible for controlling the execution of a session on the OS. Thescheduler can decide when to remove a current session from execution andcause another session to be executed. A session may comprise of a threador a group of threads on an offload processor. Depending on a number ofparameters, including such factors as the characteristics of the currentsession—whether it is stalled or running or waiting for a packet, theamount of execution resources allocated to the session, and factors suchas time allocated to the current session, the hardware scheduler maymake a context switching decision. When a packet arrives at the hardwarescheduler, and based on meeting any of the above said criteria, thescheduler may decide to context switch if the packet is for a differentsession.

As seen in FIG. 4, a method 400 can wait for packets or other data(402). Incoming packets can be received by a monitor buffer, queue orfile. Once a packet or service level specification (SLS) has beenreceived, there may be a check to ensure other conditions are met (406).If the packet/data has arrived (and optionally, other conditions met,such as those noted above) (Yes from 406), a packet session status isdetermined (408). If the packet is part of a current session (Yes from408), it can be queued for the current session (412) and processed aspart of the current session (410). In some embodiments, this can includehardware scheduler queuing the packet and sending it to an offloadprocessor for processing.

If a packet is not part of a current session (No from 408), it can bedetermined if the packet is for a previous session (414). If the packetis not from a previous session (No from 414), it can be determined ifthere is enough memory for a new session (416). If there is enoughmemory (Yes from 416), when the offload processor(s) is ready (428), thetransfer of context data can be made to the cache memory of theprocessor(s) (430). Once such a transfer is complete, the session canrun (410).

If the packet is from a previous session (Yes from 414) or there is notenough memory for a new session (No from 416), it can be determined ifthe previous session or new session is of the same color (418). If thisis not the case, a switch can be made to the previous session or newsession (420). A least recently uses (LRU) cache entity can be flushed,and the previous session context can be retrieved, or the new sessioncontext created. The packets of this retrieved/new session can beassigned a new color which can be retained. In some embodiments, thiscan include reading context data stored in a low latency memory to thecache of an offload processor. If a previous/new session is of the samecolor (Yes from 418), a check can be made to see if the color pressurecan be exceeded (422). If this is not possible, but another color isavailable (“No, other color available” from 422), a switch to theprevious or new session can be made (i.e., 420). If the color pressurecan be excluded, or it cannot, but no other color is available (“Yes/No,other color unavail.”), an LRU cache entity of the same color can beflushed, and the previous session context can be retrieved, or the newsession context created (424). These packets will retain their assignedcolor. Again, in some embodiments, this can include reading context datastored in a low latency memory to the cache of an offload processor.

In the event of a context switch (420/424), the new session can beinitialized (426). When the offload processor(s) are ready (428), thetransfer of context data can be made to the cache memory of theprocessor(s) (430). Once such a transfer is complete, the session canrun (410).

Referring still to FIG. 4, While the offload processor is processing apacket (410), there is a periodic check to see if the packet hasfinished processing (432) and return if processing is not done (No,dequeue packets from 432). If the packet is done (Yes from 432), ahardware scheduler can looks to its output queue for more packets (434).If there are more packets (Yes from 434) and the offload processor isready to receive them (Yes from 436), the packets can transferred to theoffload processor. In some embodiments, packets can be queued into theoffload processor as soon as a “ready for processing” message istriggered by the offload processor. After the offload processor is doneprocessing the packets, the entire cycle can repeat beginning with thehardware scheduler checking to which session the packet belongs, etc.

If an offload processor is not ready for a packet (No from 436) and itis waiting for rate limit (438), the hardware scheduler can check to seeif there are other packets available. If there are no more packets inthe queue, the hardware scheduler can goes into a wait mode, waiting forrate limit until more packets arrive. Thus, the hardware scheduler worksquickly and efficiently to manage and supply packets going to thedownstream resource.

As shown, a session can be preempted by the arrival of a packet from adifferent session, resulting in the new packet being processed as notedabove (406).

It should be appreciated that in the foregoing description of exemplaryembodiments of the invention, various features of the invention aresometimes grouped together in a single embodiment, figure, ordescription thereof for the purpose of streamlining the disclosureaiding in the understanding of one or more of the various inventiveaspects. This method of disclosure, however, is not to be interpreted asreflecting an intention that the claimed invention requires morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive aspects lie in less than allfeatures of a single foregoing disclosed embodiment. Thus, the claimsfollowing the detailed description are hereby expressly incorporatedinto this detailed description, with each claim standing on its own as aseparate embodiment of this invention.

It is also understood that the embodiments of the invention may bepracticed in the absence of an element and/or step not specificallydisclosed. That is, an inventive feature of the invention may beelimination of an element.

Accordingly, while the various aspects of the particular embodiments setforth herein have been described in detail, the present invention couldbe subject to various changes, substitutions, and alterations withoutdeparting from the spirit and scope of the invention.

What is claimed is:
 1. A scheduling system for a packet processingsystem, comprising: a classification circuit connected to a memory busand configurable to classify network packets, placing the classifiednetwork packets into first multiple input/output queues, a schedulingcircuit for reordering the network packets received from theclassification circuit through the first multiple input/output queuesand placing the reordered network packets into second multipleinput/output queues, an arbitration circuit for directing networkpackets received from the scheduling circuit through the second multipleinput/output queues to multiple output ports, and multiple offloadprocessors, each coupled to at least one of the multiple output ports,the offload processors configured to modify the network packets.
 2. Thesystem of claim 1 wherein the memory bus supports direct memory access,and the multiple offload processors can direct modified packets back tothe memory bus.
 3. The system of claim 1 wherein the classificationcircuit is configured to classify network packets based on sessionmetadata.
 4. The system of claim 1 wherein the scheduling circuit isconfigured to direct network packets based on availability of respectivemultiple offload processors.
 5. The system of claim 1 wherein thescheduling circuit is configured to reorder network packets according tosession priority.
 6. The system of claim 1 wherein the schedulingcircuit is configured to initiate a context switch for at least one ofthe multiple offload processors.
 7. The system of claim 1 wherein thescheduling circuit is configured to transfer network packets into adefined traffic management queue.
 8. The system of claim 1 wherein thescheduling circuit is configured to determine when network packetprocessing for each of the multiple offload processors is complete. 9.The system of claim 1 wherein the scheduling circuit is configured tooperate in a preemption mode to control session execution.
 10. Ascheduling system for a packet processing system, comprising: aclassification circuit connected to a memory bus and configurable toclassify network packets based on session metadata, and placing theclassified network packets into first multiple input/output queues, ascheduling circuit configured to reorder the network packets receivedfrom the classification circuit through the first multiple input/outputqueues and placing the classified network packets into second multipleinput/output queues, an arbitration circuit for directing networkpackets received from the scheduling circuit through the second multipleinput/output queues to multiple output ports, and multiple offloadprocessors, each coupled to one of the multiple output ports, theoffload processors configured to modify network packets and direct themodified packets to the memory bus.
 11. The system of claim 10 whereinthe scheduling circuit is configured to direct network packets based onavailability of respective multiple offload processors.
 12. The systemof claim 10 wherein the scheduling circuit is configured to reordernetwork packets according to session priority of the network packets.13. The system of claim 10 wherein the scheduling circuit is configuredto initiate a context switch for at least one of the multiple offloadprocessors.
 14. The system of claim 10 wherein the scheduling circuit isconfigured to transfer network packets into a defined traffic managementqueue.
 15. The system of claim 10 wherein the scheduling circuit isconfigured to determine when network packet processing for each of themultiple offload processors is complete.
 16. The system of claim 10wherein the scheduling circuit is configured to operate in a preemptionmode to control session execution.